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(54) Tide: IMAGE PROCESSING METHOD AND SYSTEM INVOLVING CONTOUR DETECHON STEPS 



(57) Abstract 

A method for processing an in- 
tensity image (Jo) formed of points 
representing an object (LV). includ- 
ing steps for automatically supplying 
points so as to determine a minimum 
path representing the object contour 
(CTT). involving: acquiring data of the 
intensity image, evaluating probabil- 
ity values (P) indicating the probabil- 
ity that a given point of the image 
belongs to predefined regions of the 
image relating to the object, and de- 
riving a contour model (SP) of the 
object from said probability values 
(P). transforming the image (Jo) into a 
three-dimensional matrix image (IM) 
based on said contour model (SP) and 
said probability values (P), perform- 
ing dynamic programming (DP) in or- 
der to determine said minimum path 
(CT) in said three-dimensional ma- 
trix image (IM). Application: medical 
imaging systems. 
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Image processing method and system involving contour detection steps. 



The invention relates to a method for processing an intensity image formed of 
points, which method involves steps for contour detection of an object represented in the 
image, including a step for a dynamic programming procedure. The invention also relates to a 
system with means for carrying out such a method and to an X-ray apparatus associated with 
5 such a system. 

The invention is used in the medical imaging industry. 



A method for the detection of contours in an image is already known from the 
10 publication "Encoding of a priori Information in Active Contour Models" in "IEEE 

TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 
18, NO. 9, SEPTEMBER 1 996". This publication uses the theory of active contours to model 
the problem of contour recovery as an energy minimization process. The computational 
solutions based on dynamic programming require that the energy associated with a contour 
1 5 candidate is decomposed into an integral of local energy distributions. The cited publication 
discloses a grammatical framework that models different local energy models and a set of 
allowable transitions between these models. The grammatical encoding is utilized to represent 
a priori knowledge about the shape of the object and intensity elements of the associated 
image. The variability encountered in niunerical experiments is addressed with the energy 
20 minimization procedure which is embedded in the grammatical framework. The cited 
publication discloses an algorithmic solution that combines an algorithm, denoted as the 
Knuth-Morris-Pratt algorithm for string matching with a dynamic programming algorithm for 
energy minimization. 

The method disclosed in the above publication necessarily includes an initialization step. 
25 Furthermore, the method uses only local information to represent the a priori knowledge. 



It is an object of the present invention to provide a completely automated 
method of contour detection in a complicated image, that is to say in a noisy and blurred 
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image. For example, the invention aims to process a medical image so as to detect the contour 
of the left ventricle of the heart. 

The object of the invention is achieved by means of a method as disclosed in 

claim L 

5 The method according to the invention mitigates the three major drawbacks of 

the known active contour detection method: initiaUzation dependency, exclusive use of local 
information and occlusion sensitivity. An important feature of the method of the invention is 
the introduction of high-order a priori information in each step of the method. Furthermore, 
the initialization step is completely automated by way of close cooperation between a multi- 
10 resolution neural network and a hidden Markov model-enhanced dynamic progranmiing 
procedure. 

An X-ray apparatus associated with a system for carrying out the automatic 
contour detection of the left ventricle in a digital X-ray image is also proposed. 



15 



20 



25 



These and other aspects of the invention are apparent fi-om and will be elucidated by way of 
non-limitative example, with reference to the embodiments described hereinafter and the 
accompanying drawings; therein: 

Fig. 1 shows diagrammatically a left ventricle with anatomic classes; 

Fig.2 shows a diagram of the feature vector computations for the neural 



network 



image 



method. 



Fig.3 shows the rubber band in an original X-ray image of the left ventricle ; 
Fig.4 shows the transformed image of Fig.3 ; 
Fig.5 shows a part of the matrix image ; 

Fig.6 shows the left ventricle contour detected and superimposed on the original 
Fig. 7 shows an X-ray apparatus associated with a system for carrying out the 



30 



The invention relates to contour detection in low-contrast and noisy images 
with a wide intrinsic variability as commonly encountered in medical applications. The 
invention also relates to a robust automatic procedure for application in industrial apparatus. 
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The invention will be described hereinafter as an example of an application in 
the medical imaging field. It consists of a method for automatically detecting the left ventricle 
LV boundaries in digital X-ray right anterior oblique (RAO) projection ventriculograms (X- 
ray images of a ventricle). 

As disclosed by the cited publication, it is known to model the problem of 
contour recovery as an energy minimization process. An energy function is associated with 
every candidate contour. These active contours are controlled by internal and external forces. 
The internal forces set the overall shape characteristics such as smoothness whereas external 
forces attract the contour towards local image features such as a high gradient value. The 
minimization is carried out by a discrete dynamic progranuning DP procedure. This technique 
is referred to as active contour AC detection. The method is iterative. 

The known method has three main drawbacks. The first one is a low precision 
of initial contour. A problem inherent of each iterative method is the initialization phase. This 
is even more cmcial for the active contour method which requires a good initial contour in 
order to be able to converge towards the desired edges. This contour usually has to be 
manually drawn by an operator. 

The second drawback lies in the form of the external force that is usually 
derived from local information only. Any a priori knowledge which could help is thus difficult 
to introduce. According to the cited publication, this problem is solved by imposing 
constraints on the overall shape of the desired contour which restrict the internal force inside a 
small set of admissible forms. 

The third drawback lies at the core of the AC algorithm which usually is not 
robust to occlusion and fails when the pre-processing edge detection step erroneously skips 
parts of the desired contour. 

In order to overcome these difficulties, the invention proposes a new method 
which is based on close cooperation between a multi-resolution neural network and an 
enhanced dynamic programming procedure. 

This method includes a first step for initialization which involves the use of a 
neural network NN for automatically finding an initial contour of the left ventricle in the 
original image. This NN uses global information derived from a pyramid of filters, and the 
resultant posterior probabilities are used as basic states for a second step for a dynamic 
programming procedure DP. 

Digital X-ray imaging is commonly used in cardiology applications. A frequent 
type of heart examination consists in injecting a contrast medium in the left ventricle LV of 

995206aAl _!_> 
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the patient through a catheter This allows the visualization of the LV chamber in X-ray 
images. An image or an image sequence may be recorded, and the physician has to outline the 
LV boimdaries in the end-systolic and end-diastolic frames. Relevant parameters, such as LV 
chamber volumes, ejection fraction and stroke volume can then be computed. Nowadays, 
5 techniques for semi-automatic segmentation of the LV in the relevant image frames and for 
estimation of its volume are available. However, the state-of-the-art algorithms require a fair 
degree of interaction with a human operator so as to at least pin-point key points in each of the 
image frames. 

Automating this task is challenging, because local image configurations do not 

1 0 provide sufficient information to avoid false key identifications and because the gray scale 
ventriculograms have a poor contrast and a high level of noise. This noise is due to the 
scattering of radiation by tissues not related to the ventricle, artifacts generated by the patient's 
breathing during the catheterization procedure, and interference of the ribs and the diaphragm 
with the LV. Moreover, the injected contrast medium does not mix uniformly with blood in 

1 5 the LV and the apex zone of the LV generally does not receive much dye. 

A major problem with an active contour framework is that its requires a good 
initial guess in order enable convergence to the desired position. In the medical field this 
initial contour is usually manually drawn by the operator. Since the aim of the invention is to 
propose a fully automatic method in order to build an automated system, the first step of the 

20 method has to provide a reasonable initial contour. This object is achieved in said first step on 
the basis of training an artificial neural networks NN with examples. 

This first step is based on a NN that uses multi-resolution information to 
estimate the probabilities that a given point belongs to predefined classes. This first step 
includes several sub-steps. 

25 In a first sub-step, the NN stmcture is defined, i.e. its output (feature classes), the points that 
should be selected and the NN input vectors computed for these points. In a second sub-step, 
the NN training and results are described, and in a third sub-step the way of building an initial 
AC contour from its output is given. 

The first sub-step is to define the "desired classes" corresponding to the specific 

30 features of the object to be detected in the original image. For application to left ventricle 

contour detection, 15 anatomic classes have been defined accordingly as shown in Fig.l. The 
classes BPn and BAn are the LV boundaries (with n=l,..,4). The three key points are the 
antero and postero sides of the aortic valve, denoted VA and VP, respectively, and the apex, 
denoted AP. These points are very important for characterizing the LV chamber and the 
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physician is usually required to select them manually. In order to increase the selectivity of the 
neural network, additional classes, being non-LV-boundaries classes, are defined as PA, PP, 
CA, CI and CE. 

In order to reduce the computation time, only the points with "maximum 
5 gradient" are fed to the neural network. These are the points whose gradient intensity is a local 
extremimi along its direction. Points where the gradient is lower than that of the nearest 
neighbors are discarded. 

The first sub-step also serves to define neural network input vectors. Referring 
to Fig.2, starting fi-om an original image Jo, for instance of 512x512 pixels, a set of low pass 
1 0 filtered versions is generated so as to yield a multi-resolution representation. In addition to the 
finest scale corresponding to Jo, K other scales (L*" with k=l..K) are computed a with half 
and a quarter nominal image size denoted Ji and Jj and smoothed with kernels of sizes 
(spatial standard deviation) ranging fi-om 3 to 70 pixels; this is illustrated by the pyramid of 
filters F shovra in Fig.2. Recursive filtering techniques are used to efficiently implement such 
1 5 large kernel sizes. In the implementation, much attention is paid to achieving nearly isotropic 
impulse responses. Subsequently the derivatives of the L*^ images with respect to x and y are 
computed. A controllable representation of the filtered output can be derived as a linear 
combination of the various partial derivatives. This controllable filter bank provides an 
equivalent representation of the following derivatives: 

20 Dli} =(aVax'"a^-"*)L'^ 

For the finest scale (k=0), one stops at the order 1=1, thus generating three components. For the 
other scales k=l,..,7, derivatives up to the 4^*" order are computed, leading to 105 feature 
values. To those 105 + 3 = 108 features, information about the gradient is added as illustrated 
by J' in Fig.2 : one entry for the gradient intensity and 16 binary entries to encode the gradient 
25 direction (with a single entry set to 1, the rest being set to 0). The last feature is the edge 
curvature computed for the first half-size filtered image Jl of the pyramid. 

A feature vector V(x,y) of dimension 126 is thus computed for each selected 
point at position (x,y) of the image. The use of different scales enables combination of local 
with contextual information. The neural network NN fed with this input vector V(x,y), gives 
30 the probabilities of 15 classes, denoted Ci, for the current point (x,y). The feature vector 
computations are illustrated in FIG.2. 

The second sub-step includes the multi-resolution NN training. The neural 
network architecture is thus defined by 126 input nodes, 50 hidden neurons and 1 5 output 
neurons. The training of the neural network is done with a leaming set consisting of several 
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thousands of examples (pairs of "feature vector"-"desired output") from several tens of images 
and the generalization is checked by utilizing a test set consisting of several thousands of 
examples extracted from several tens of other images. The performance of the neural network 
are of the order of 93.09% and 79.34% correct classifications for the training and testing sets 
5 respectively. 

The third sub-step concems the estimation of the neural Network Outputs. The 
outputs of the neural network are, therefore an estimate of the posterior probabilities 
P ( CiA^(x,y) ) that the selected point at position (x,y) belongs to the predetermined classes. 
The NN robustly enables the finding of the most probable region of the image for the 

1 0 important classes. For each of those classes a respective probability image is built, in which 
the pixel intensities are the corresponding probability estimated by the NN. A zero intensity is 
attributed to non-selected points. 

This image is then smoothed by a rather small kernel (spatial standard deviation 
of 3 pixels) in order to find regions with high probabilities. In these probability maps the 

15 maximum pixel intensity is found for 5 specific probability images. Actually, the first three 
maxima separated by at least 50 pixels are computed. This leads to 3^=243 possible polygon 
lines linking these 5 specific successive classes: i.e. VA, BA2, AP, BP3 and VP as shown in 
Fig.l . Simple rules are used to discard incoherent relative positions of the above five key 
points and to sort out the acceptable polygons. For example: "the apex should lie to the right 

20 of both aortic valve extremities". 

As shown in Fig.3, the neural network can thus provide a polygon line PL 
which is a very crude model of the object to be detected. This line is computed from non-local 
information and is consistent with high-order prior knowledge about the "average" object 
configuration. 

25 Fig.3 illustrates the results of the NN for an image where the diaphragm 

intersects the left ventricle. The disks are the points provided by the physician and the dashed 
line is the cmde model. The crosses along this line are the detected key points. For this image, 
the average error over the three key points is 16.9 pixels. For 58 test images of size 512 x 512, 
the detected key points lie at an average distance of 26.7 pixels from the corresponding points 

30 defined by the physician (standard deviation of 19.7). At the present time, the precision 

required in processing medical images is far greater than in the past. So it no longer suffices to 
initialize the method by only manually pointing three key points in the image. This is the 
reason why the object of the present invention is to detect the whole LV contour 
automatically. Hence, the usefiil neural network outputs are the polygon line and the 



BNSDOCtD: <WO 995206eA1_l_> 



• . wo 99/52068 PCT/IB99/005I3 

7 

probability maps which will be used to transform the image and to direct the dynamic 
programming DP procedure, respectively. 

The method according to the invention then includes a step for Image 
Transformation. Because standard DP works in 1-D, a causality rule for images has to be 

5 determined. This procedure is carried out by building an image which follows the line 

previously found with the NN. The new image is a *'rubber" band RB around this line. Fig.3 
shows the rubber band along the line defined by the NN. The image to be transformed is 
delineated around a "spline" SP mathematically calculated on the basis of the polygon line PL. 
The transfomiation is based on a minimum distance paradigm. The rubber-band image has to 

1 0 be mapped into a rectangular image. Points in overlapping regions are transformed to the 

nearest middle line. The corresponding transformed image is shovm in FIG.4. Since it is not a 
bijective transformation, some points in the image remain black, meaning that they do not 
correspond to any point. 

This rubber-band RB unfolding transformation is performed on the posterior 

1 5 probability P maps computed by the neural network so as to build a 3D probability matrix. The 
horizontal dimension is the "time" t or the path along the spline SP, the vertical one is defined 
by the width W of the rubber-band and encodes the candidate points and the third dimension 
stores the different posterior probabihties P for these points. FIG.5 shows a small part of such 
an artificial 3D probability matrix image IM. 

20 The method according to the invention also includes a step for dynamic 

programming. This step is based on an algorithm in a 2-D fomi in which, instead of merely 
linking points as known to those skilled in the art, the algorithm carries out the linking of 
couples of point and class. The cost of such a link between class i of current point S(x,y) and 
class j of a possible predecessor S' is defined as: 

2 5 c{Sj ' , 5. ) = C* {Sj ) accumulated cost 

+ p d(s \ s) point distance 
+ aC(Si) local cost 
+ y c, (j, i) j-i transition cost 
where a, p , and y are weighting coefficients. So the cost of a link depends on the 
30 accumulated cost for reaching S'j, the distance between the two points S' and S (computed in 
the original image), the cost for going through : 

S, C(S,)=-logP(Q/V(x, y)) 
and a transition cost for going fi'om class j to class i : 
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c.a,i)=-iogp(Q(t)/q(t-i)) 

where P (C^ / V(x, y)) is the posterior probability for class Ci at the point S(x,y) and is 
estimated by the multi-resolution neural network. The transition probabilities 
p(Ci(t)/Cj(t -l)) may be fixed beforehand or estimated by a system. 

5 Fig.5 shows a small synthesized sample of the resultant image matrix IM with 

the probabilities P(ci) of the various border classes for some selected points indicated in gray : 
the darker, the higher. Note that the number of points per column 

s, s- 1 , s-2. . . is not constant and that these valid points are not evenly distributed along a 
column along the time axis. 
1 0 According to the invention, the method provides the possibility to link points 

which do not lie in adjacent columns of the transformed image. These "jumps" ensure that a 
Hnk will be found even should the selection phase have erroneously missed some intermediate 
points. Previous points are thus searched for in a sector covering several tens of positions, for 
instance 51 positions. The best cost for arriving at class i of point S is defined as: 

15 c*(s,)=min[c(s;,s,)] 

j.S' 

where j e { 1 . . ..N}, and S*€ Pr ed(s) is the set of valid points in the sector area and N the 
number of predefined classes. Such links are computed for all couples (point,class) going 
forward (in the causal order) in the image matrix. The most probable contour CT is then 
retrieved as the one ending with the lowest cost. 
20 Fig.6 shows the boundary CT found by this enhanced dynamic procedure (the 

black line). It may be seen that the use of the posterior probabilities gives very accurate 
results. 

Automatic contour extraction is a difficuh issue which is not only usefiil in 
medical applications but may also be used in numerous other applications such as object based 

25 encoding (as in MPEG-4). The method according to the invention is based on the steps of 
introducing high order information in the active contoxu* detection procedure. A multi- 
resolution neural network is used to compute posterior probabilities for predefined classes. 
These probabilities are first used to define a causality and an image transfomiation. Next they 
are used to guide the dynamic programming procedure both in lowering the cost of going 

30 through points reliably classified as borders and in defining the cost for linking two such 
points. This linking is encoded into a probabilistic state automata. 
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Such fully automatic contour detection can be achieved even in the case of 
comphcated images. The computation time can be reduced by selecting the most relevant 
features in the large input vector of the NN. Accuracy can be increased by using more images 
for the NN training. 

5 Fig. 7 shows, by way of example, an X-ray examination apparatus which 

includes an X-ray source 1, a table 2 for accommodating a patient, an image intensifier device 
3 which is coupled to a camera tube 4 which applies data to a digital image processing system 
5 which includes a processor provided with means 30 which are suitable to form an image 
processing system. This image processing system 30 carries out the contour detection method 
10 according to the invention. The processor has several outputs, one output 16 of which is 
coupled to a display system 7 for the visualization of the radiographic images or intensity 
images. The display system can include storage means for storing images. 
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1 . A method for processing an intensity image formed of points representing an 

object (LV), including steps for automatically supplying points determining a minimal path 
representing the object contour (CT), involving: 

acquiring data of the intensity image (Jo), 
5 evaluating probability values (P) indicating the probability that a given point of 

the image belongs to predefined regions of the image relating to the object, and deriving a 
contour model (SP) of the object from said probability values (P), 

transforming the image (Jo ) into a three-dimensional matrix image (IM) based 
on said contour model (SP) and said probability values (P), 



1 0 performing dynamic programming (DP) in order to determine said minimum 

path (CT) in said three-dimensional matrix image (IM). 

2. A method as claimed in claim 1, in which the step of image transformation 

includes: 

1 5 building a rubber-band-shaped image which follows the contour model (SP) of 



the object (LV) derived firom said probability values, 

mapping the rubber-band-shaped image (RB) into a rectangular mbber-band 
image having the contour model (SP) straightened as a line for the longitudinal axis (t) and a 
given distance (W) of the points of the image to said longitudinal axis for the orthogonal axis, 
20 forming a three dimensional matrix image based on the rectangular mbber-band 

image by associating a third dimension with the mbber-band image constituted by the 
probability values of the points. 

3. A method as claimed in claim 2, in which the dynamic programming includes: 

25 imparting a causal orientation to said longitudinal axis (t), 

and, in the 3-D matrix image linking a point (S) associated with its probability 
value to a predecessor (S*) associated with its probability value while minimizing a cost 
function taking into account said probability values and local and contextual intensity 
information relating to the points to link. 
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4. A method as claimed in one of the claims 1 to 3, in which the probability 
values are provided by a neural network which has inputs for vectors taking into account local 
and contextual intensity information relating to points of the original image and outputs for 
providing the probability values indicating the probability that a point belongs to a predefined 

5 region of the original image. 

5. A method as claimed in claim 4, in which a number of important points are 
selected fi-om among the points determined by the neural network as having a high probability 
of belonging to corresponding regions, said important points determining a polygon (PL) in 

10 the original image, and in which a mathematical line (SP), called spHne, is built from the 
polygon (PL) so as to constitute a contour model (SP) for the object (LV) in the image . 

6. A method as claimed in one of the claims 1 to 5 where the processed image is a 
medical X-ray digital image, the object is the left ventricle of the heart, and the object contour 

1 5 is the left ventricle contour which includes at least three important points constituted by the 
aortic valve limits (VA, VP) and the apex (AP). 

7. A system for the processing of a medical image including means (5,30) for 
carrying out a method as claimed in one of the claims 1 to 6. 

20 

8. An X-ray apparatus which includes an X-ray detector for forming a medical 
digital image, said apparatus being associated with an image processing system as claimed in 
claim 7 and with a system (7) for displaying the processed image. 
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